Image capturing device and image processing device, control methods of the same, and storage medium

ABSTRACT

An image capturing device includes image sensor, capturing a first image in a first image capturing region, capturing a second image in the first image capturing region at a shorter exposure time than that of the first image, after capturing the first image, capturing a third image in a second image capturing region at a shorter exposure time than that of the first image, after capturing the second image; and capturing a fourth image in the second image capturing region at a longer exposure time than those of the second and third images, after capturing the third image; and combining unit configured to carry out alignment processing between the images using the second image and the third image, and combine the first image and the fourth image on the basis of a result of the alignment processing.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a technique for obtaining a wider-rangeimage of a starry sky, i.e., a panoramic starry sky image, by combiningimages captured continuously while successively changing the shootingdirection so that regions which overlap with each other appear.

Description of the Related Art

shooting is known as one shooting method for capturing a wide range of astarry sky in a single image. Japanese Patent Laid-Open No. 2005-328497discloses the following shooting method as an example of such a method.A plurality of unit images, each of which constitutes a part of a rangeto be shot, are captured while successively changing the shootingdirection. Image regions of a predetermined size are then cut out fromthe capturing unit images so that regions which overlap with each otherare produced, and a panoramic image is then generated by superimposingthe cut-out image regions in sequence.

Problems arising if this method is applied when shooting a panorama of astarry sky will be described next. When shooting a starry sky, there isonly an extremely small amount of light from the stars, and thus longexposures, such as 30 seconds or 1 minute, are often used. Astronomicalbodies exhibit diurnal motion in accordance with the earth's rotation,and thus stars exposed for a long time will appear not as points oflight, but rather as tracks of light.

When shooting a panorama of a starry sky, it is necessary to generatethe panoramic image by shooting images at different shooting directions,at exposure times that are short enough to avoid making the stars appearas tracks of light, and then stitch the images together. There are alsosituations where one wishes to shoot a long-exposure panorama of astarry sky. The stars move over time, and will thus be in differentpositions from image to image, which makes it difficult to successfullyposition the images with respect to each other. FIGS. 8A-8C illustratean example in which, when shooting a panorama of a starry sky,positioning fails when combining two images shot from differentdirections. FIG. 8A illustrates the first shot image, where 801indicates the background. FIG. 8B illustrates the second shot image,where 802 indicates the background. With the passage of an amount oftime equivalent to a combination of the time of the long exposureshooting of the first image and time taken by the user to set the imagecapturing device to change the shooting direction, the stars are in adifferent position in the second image compared to the first image. FIG.8C illustrates a state in which the first and second shot images havebeen positioned using the stars as a reference, resulting in thebackgrounds 801 and 802 being combined having been shifted from eachother by an amount equivalent to the stars' movement.

On the other hand, Japanese Patent Laid-Open No. 2016-005160 discloses atechnique in which optical shake correction means are used to correctpositional skew in the image capturing plane, caused by the diumalmotion of the astronomical bodies. Images are repeatedly shot andcombined to obtain a shot image with a wider angle of view, while at thesame time ensuring that the positioning succeeds.

However, the conventional technique disclosed in Japanese PatentLaid-Open No. 2016-005160 uses optical shake correction means, and thereis thus a problem in that the maximum value of the change in shootingdirection is limited. This means that it is not possible to capture animage of an astronomical body having an even wider angle of view.

SUMMARY OF THE INVENTION

Having been achieved in light of the above-described problem, thepresent invention provides an image capturing device capable ofcapturing a high-quality panoramic image even when the position of astar, which serves as a subject, has changed between images shot fromdifferent directions.

According to a first aspect of the present invention, there is providedan image capturing device capable of generating a single combined imageby combining a plurality of images, each image being from a differentimage capturing region, and each image having a region at leastpartially shared by another image capturing region, the devicecomprising: an image sensor configured to capture a subject image, theimage sensor: capturing a first image in a first image capturing region;capturing a second image in the first image capturing region at ashorter exposure time than that of the first image, after capturing thefirst image; capturing a third image in a second image capturing regionat a shorter exposure time than that of the first image, after capturingthe second image; and capturing a fourth image in the second imagecapturing region at a longer exposure time than those of the second andthird images, after capturing the third image; and the device furthercomprising: at least one processor or circuit configured to function asthe following unit: a combining unit configured to carry out alignmentprocessing between the images using the second image and the thirdimage, and combine the first image and the fourth image on the basis ofa result of the alignment processing.

According to a second aspect of the present invention, there is providedan image processing device capable of generating a single combined imageby combining a plurality of images, each image being from a differentimage capturing region, and each image having a region at leastpartially shared by another image capturing region, the devicecomprising: at least one processor or circuit configured to function asthe following units: an obtainment unit configured to obtain a firstimage captured in a first image capturing region, a second imagecaptured in the first image capturing region at a shorter exposure timethan that of the first image after the first image has been captured, athird image captured in a second image capturing region at a shorterexposure time than that of the first image after the second image hasbeen captured, and a fourth image captured in the second image capturingregion at a longer exposure time than those of the second and thirdimages after the third image has been captured, the images beingcaptured by image sensor; and a combining unit configured to carry outalignment processing between the images using the second image and thethird image, and combine the first image and the fourth image on thebasis of a result of the alignment processing.

According to a third aspect of the present invention, there is provideda method of controlling an image capturing device capable of generatinga single combined image by combining a plurality of images, each imagebeing from a different image capturing region, and each image having aregion at least partially shared by another image capturing region, themethod comprising: capturing a first image in a first image capturingregion; capturing a second image in the first image capturing region ata shorter exposure time than that of the first image, after capturingthe first image; capturing a third image in a second image capturingregion at a shorter exposure time than that of the first image, aftercapturing the second image; capturing a fourth image in the second imagecapturing region at a longer exposure time than those of the second andthird images, after capturing the third image; and carrying outalignment processing between the images using the second image and thethird image, and combining the first image and the fourth image on thebasis of a result of the alignment processing.

According to a fourth aspect of the present invention, there is provideda method of controlling an image processing device capable of generatinga single combined image by combining a plurality of images, each imagebeing from a different image capturing region, and each image having aregion at least partially shared by another image capturing region, themethod comprising: obtaining a first image captured in a first imagecapturing region, a second image captured in the first image capturingregion at a shorter exposure time than that of the first image after thefirst image has been captured, a third image captured in a second imagecapturing region at a shorter exposure time than that of the first imageafter the second image has been captured, and a fourth image captured inthe second image capturing region at a longer exposure time than thoseof the second and third images after the third image has been captured,the images being captured by image sensor; and carrying out alignmentprocessing between the images using the second image and the thirdimage, and combining the first image and the fourth image on the basisof a result of the alignment processing.

According to a fifth aspect of the present invention, there is provideda non-transitory computer-readable storage medium storing a program forcausing a computer to execute the steps of a method of controlling animage capturing device capable of generating a single combined image bycombining a plurality of images, each image being from a different imagecapturing region, and each image having a region at least partiallyshared by another image capturing region, the method comprising:capturing a first image in a first image capturing region; capturing asecond image in the first image capturing region at a shorter exposuretime than that of the first image, after capturing the first image;capturing a third image in a second image capturing region at a shorterexposure time than that of the first image, after capturing the secondimage; capturing a fourth image in the second image capturing region ata longer exposure time than those of the second and third images, aftercapturing the third image; and carrying out alignment processing betweenthe images using the second image and the third image, and combining thefirst image and the fourth image on the basis of a result of thealignment processing.

According to a sixth aspect of the present invention, there is provideda non-transitory computer-readable storage medium storing a program forcausing a computer to execute the steps of a method of controlling animage processing device capable of generating a single combined image bycombining a plurality of images, each image being from a different imagecapturing region, and each image having a region at least partiallyshared by another image capturing region, the method comprising:obtaining a first image captured in a first image capturing region, asecond image captured in the first image capturing region at a shorterexposure time than that of the first image after the first image hasbeen captured, a third image captured in a second image capturing regionat a shorter exposure time than that of the first image after the secondimage has been captured, and a fourth image captured in the second imagecapturing region at a longer exposure time than those of the second andthird images after the third image has been captured, the images beingcaptured by image sensor; and carrying out alignment processing betweenthe images using the second image and the third image, and combining thefirst image and the fourth image on the basis of a result of thealignment processing.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating the configuration of a firstembodiment of an image capturing device according to the presentinvention.

FIGS. 2A and 2B are conceptual diagrams illustrating a panoramiccombination of a plurality of shot images.

FIG. 3 is a flowchart illustrating normal shooting operations.

FIGS. 4A and 4B are flowcharts illustrating operations for shooting apanorama of a starry sky, according to the first embodiment.

FIGS. 5A and 5B are data flow diagrams illustrating operations accordingto the first embodiment.

FIGS. 6A and 6B are flowcharts illustrating operations for shooting apanorama of a starry sky, according to a second embodiment.

FIGS. 7A and 7B are conceptual diagrams illustrating a warning screenaccording to a third embodiment.

FIGS. 8A to 8C are conceptual diagrams illustrating an issue arisingduring panoramic combination for a starry sky.

FIG. 9 is a flowchart illustrating operations for shooting a panorama ofa starry sky, according to a fourth embodiment.

FIG. 10 is a conceptual diagram illustrating the flow of the generationof a panoramic image, according to a fourth embodiment.

FIG. 11 is a flowchart illustrating operations for shooting a panoramaof a starry sky, according to the fourth embodiment.

FIG. 12 is a flowchart illustrating operations in a panoramic starry skyimage generation process, according to the fourth embodiment.

FIG. 13 is a flowchart illustrating operations in a panoramic starry skyimage generation process, according to a fifth embodiment.

FIG. 14 is a conceptual diagram illustrating the flow of the generationof a panoramic image, according to the fifth embodiment.

FIG. 15 is a conceptual diagram illustrating a warning screen accordingto the fifth embodiment.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, embodiments of the present invention will be described indetail with reference to the appended drawings.

First Embodiment

FIGS. 2A and 2B are diagrams illustrating an overview of panoramicshooting. In the present embodiment, panoramic shooting is realized byshooting while continuously changing the capturing direction of an imagecapturing device 201, which is done manually by a user 202 or by anautomatic tracking mount or the like, as illustrated in FIG. 2A. Asillustrated in FIG. 2B, a plurality of images are shot so that commonregions of a subject are present in parts of each of the shot images.Feature points are then extracted from the common regions of the images,and a motion vector indicating the extent to which those feature pointshave moved is detected. An affine transformation coefficient, forexample, is then calculated from the motion vector, and the two imagesare then superimposed so that the feature points coincide. This producesan image in which parts aside from the common regions have beenextended. Repeating these multiple times makes it possible to generate apanoramic image having a wider angle of view than the angle of viewachieved when shooting a single image.

FIG. 1 is a block diagram illustrating the configuration of a firstembodiment of an image capturing device according to the presentinvention. In FIG. 1, an image capturing device 100 includes a shootinglens 101, which forms a subject image, and an autofocus (AF) drivecircuit 102, which adjusts the focus of the shooting lens 101. The AFdrive circuit 102 is constituted by a DC motor, a stepping motor, or thelike, for example, and adjusts the focus by changing the position of afocus lens in the shooting lens 101 under the control of a microcomputer123.

The shooting lens 101 includes an aperture stop 103, and the aperturestop 103 is driven by an aperture drive circuit 104. An optical aperturevalue is calculated by the microcomputer 123, and the amount by whichthe aperture drive circuit 104 drives the aperture stop 103 isdetermined on the basis of that value.

A main mirror 105 is arranged behind the aperture stop 103. The mainmirror 105 switches between a state in which a light beam passingthrough the shooting lens 101 is guided to a viewfinder or to an imagesensor 112. The main mirror 105 is normally arranged in a position thatreflects the light beam upward so that the light beam is guided to theviewfinder, but flips upward, out from the optical path, when shootingor executing a live view display, so that the light beam is guided tothe image sensor 112. Note that the main mirror 105 is a half mirror,the central part of which allows a small amount of light to pass. Somelight is therefore allowed to pass and is guided to a focus detectionsensor (not shown) for the purpose of focus detection. A defocus amountof the shooting lens 101 is found by computing the output of this focusdetection sensor. The microcomputer 123 evaluates the computation resultand instructs the AF drive circuit 102 to drive the focus lens.

The main mirror 105 is driven upward and downward by a mirror drivecircuit 107, in response to instructions from the microcomputer 123. Asub mirror 106 is arranged behind the main mirror, and reflects thelight beam passing through the main mirror 105 so as to guide that lightbeam to the aforementioned focus detection sensor. The light beam thathas passed through the central part of the main mirror 105 and beenreflected by the sub mirror 106 is also incident on an exposure amountcalculation circuit 109, and reaches a photometry sensor for the purposeof photoelectric conversion, which is disposed within the exposureamount calculation circuit 109.

A pentaprism, which partially constitutes the viewfinder, is arrangedabove the main mirror 105. The viewfinder is also constituted by afocusing plate, an eyepiece lens (not shown), and the like.

A focal plane shutter 110, which opens and closes the optical path ofthe shooting lens 101, is driven by a shutter drive circuit 111. Thetime for which the focal plane shutter 110 is open is controlled by themicrocomputer 123.

The image sensor 112 is arranged behind the focal plane shutter 110. ACCD, a CMOS sensor, or the like is used for the image sensor 112, andconverts the subject image formed by the shooting lens 101 into anelectrical signal. The output from the image sensor 112 is input to anA/D converter 115. The A/D converter 115 converts analog output signalsfrom the image sensor 112 into digital signals.

An image signal processing circuit 116 is realized by a logic devicesuch as a gate array. The image signal processing circuit 116 includes aluminance adjustment circuit 116 a, a gamma correction circuit 116 b, amovement amount calculation circuit 116 c, a positioning circuit 116 d,a geometric conversion circuit 116 e, and a magnification circuit 116 f.The image signal processing circuit 116 further includes a trimmingcircuit 116 e, a combining circuit 116 j, a developing circuit 116 k,and a compression/decompression circuit 116 l.

The luminance adjustment circuit 116 a adjusts the brightness usingdigital gain. The gamma correction circuit 116 b adjusts the luminanceusing gamma characteristics. The movement amount calculation circuit 116c calculates a movement amount in a plurality of images. The positioningcircuit 116 d positions the plurality of images in accordance with themovement amount in the images. The geometric conversion circuit 116 ecorrects for the curvature of the shooting lens 101. The magnificationcircuit 116 f changes the size of the images. The trimming circuit 116 ecuts out parts of the images. The combining circuit 116 j combines theplurality of images. The developing circuit 116 k develops the imagedata. The compression/decompression circuit 116 l converts the imagedata into a typical image format such as JPEG.

A display drive circuit 117, a display member 118 that uses TFTs,organic EL, or the like, a memory controller 119, memory 120, anexternal interface 121 for connectivity with a computer or the like, andbuffer memory 122 are connected to the image signal processing circuit116.

The image signal processing circuit 116 carries out filtering, colorconversion, and gamma processing, as well as compression processingaccording to the JPEG format, on the digitized image data, and outputsthe result to the memory controller 119. At this time, the image beingprocessed can also be stored in the buffer memory 122 temporarily.

The image signal processing circuit 116 can also output image signalsfrom the image sensor 112, image data that conversely has been inputfrom the memory controller 119, and the like to the display member 118through the display drive circuit 117. These functions are switched inresponse to instructions from the microcomputer 123.

The image signal processing circuit 116 can also output information,such as exposure or white balance information of the signal from theimage sensor 112, to the microcomputer 123 as necessary. Themicrocomputer 123 makes instructions pertaining to white balanceadjustment, gain adjustment, and the like on the basis of thatinformation.

In continuous shooting operations, shot data is first stored in thebuffer memory 122 in an unprocessed state. The unprocessed image data isthen read out through the memory controller 119 and subjected to imageprocessing, compression processing, and the like by the image signalprocessing circuit 116 to carry out the continuous shooting. The numberof continuous shots depends on the capacity of the buffer memory 122 or,when shooting a panorama, the image size. The memory controller 119stores the unprocessed digital image data input from the image signalprocessing circuit 116 in the buffer memory 122, and stores theprocessed digital image data in the memory 120. It is also possible toconversely output image data from the buffer memory 122, the memory 120,or the like to the image signal processing circuit 116. There are alsocases where the memory 120 can be removed. Note that the memorycontroller 119 can also output images stored in the memory 120 to theexterior through the external interface 121, which enables a computer orthe like to be connected.

Operation members 124 communicate their state to the microcomputer 123,and the microcomputer 123 controls the respective constituent elementsin accordance with changes in the operation members. A switch SW1 (125)and a switch SW2 (126) are switches that turn on and off when a releasebutton is operated, and each is one input switch in the operationmembers 124.

A state where only the switch SW1 (125) is on corresponds to a statewhere the release button is depressed halfway. Autofocus operations andphotometry operations are carried out in this state. A state in whichboth the switches SW1 (125) and SW2 (126) are on corresponds to a statewhere the release button is fully depressed. This is a state where arelease switch for recording an image is on. Shooting is carried out inthis state. Continuous shooting operations are carried out while theswitches SW1 (125) and SW2 (126) remain on.

The following switches, which are not shown, are also connected to theoperation members 124: an ISO setting button; a menu button; a setbutton; a flash settings button; a single shot/continuousshooting/self-timer switching button; a movement + (plus) button and amovement − (minus) button for moving through menus and images to beplayed back; an exposure correction button; a displayed image enlargebutton; a displayed image reduce button; a playback switch; an aperturebutton for bringing the aperture stop 103 to the set aperture value; adelete button for deleting shot images; information display buttonspertaining to shooting, playback, and the like; and so on. The states ofthese switches are detected. Assigning the functions of theaforementioned plus button and minus button to a rotary dial switchmakes it possible to select numerical values, functions, and the likemore easily.

A liquid crystal drive circuit 127 causes operational states, messages,and the like to be displayed in an external liquid crystal displaymember 128, an in-viewfinder liquid crystal display member 129, and thelike using text and images, in response to display commands from themicrocomputer 123. A backlight (not shown), which uses LEDs or the like,is provided in the in-viewfinder liquid crystal display member 129, andthe LEDs are also driven by the liquid crystal drive circuit 127.

The microcomputer 123 can calculate the remaining number of shots thatcan be taken, having confirmed the memory capacity through the memorycontroller 119, on the basis of predictive value data for the image sizeaccording to the ISO sensitivity, image size, and image quality setbefore shooting. This information can also be displayed in the externalliquid crystal display member 128 and the in-viewfinder liquid crystaldisplay member 129 as necessary.

Non-volatile memory (EEPROM) 130 can store data even when the camera isnot turned on. A power source unit 131 supplies the necessary power tothe various ICs, drive systems, and the like. An internal clock 132measures the passage of time, and can save shooting times and the likein image files recorded into the memory 120, superimpose the shootingtime on images themselves (as will be described later), and so on. Agyrosensor 133 detects the angular velocity of rotation of the imagecapturing device 100 on two or three axes. An azimuth indicator 134detects the direction in which the image capturing device is facing.

Operations of the image capturing device configured as described abovewill be described next. FIG. 3 is a flowchart illustrating shootingoperations by the image capturing device according to the firstembodiment.

First, before starting the shooting operations, the exposure amountcalculation circuit 109 calculates the exposure amount, and the aperturevalue, accumulation time, and ISO sensitivity are set. The shootingoperations are carried out upon the switch SW2 (126) being depressed bythe user.

In step S301, the microcomputer 123 notifies the aperture drive circuit104 of the predetermined aperture value, and the aperture stop 103 isadjusted to the target aperture value. Power is supplied to the imagesensor 112, the A/D converter 115, and the like to prepare for shooting.Once the preparations are complete, the mirror drive circuit 107 isdriven to flip the main mirror 105 up, so that the subject image isincident on the image sensor 112. A shutter drive circuit opens a frontcurtain (not shown) of the focal plane shutter 110 so that the subjectimage is incident on the image sensor 112. Then, after a predeterminedaccumulation time, a rear curtain (not shown) of the shutter 110 isclosed so that light enters the image sensor 112 only for theaccumulation time. Exposure is carried out through this sequence ofoperations.

In step S302, an image signal is read out to the image signal processingcircuit 116 through the A/D converter 115 and stored in the buffermemory 122. In step S303, the read-out image signal is developed by thedeveloping circuit 116 k and converted into image data. At this time,image processing such as white balance processing, gamma processingcarried out by the gamma correction circuit 116 b to apply gain to darkparts, and the like may be used to bring to the image to an appropriateimage quality.

In step S304, the obtained image data is converted into a generic dataformat, such as JPEG, by the compression/decompression circuit 116 l. Instep S305, the converted image data is saved into the memory 120, whichis an SD card or Compact Flash (registered trademark). This ends theshooting operations.

Note that in step S303, rather than carrying out the image processing,developing processing, and so on, the read-out image signal may belosslessly compressed directly in step S304, and may then be saved in astorage medium in step S305. The switch can be made by the user, usingthe operation members 124.

A starry sky panorama shooting mode will be described next. Although astarry sky panorama can be shot in a mode that shoots images whileshifting the image capturing device in the horizontal direction or amode that shoots images while shifting the image capturing device in thevertical direction, an example of shooting while shifting in thehorizontal direction will be described here.

When the user uses the operation members 124 to set the starry skypanorama shooting mode, power is supplied to the image sensor 112 andthe A/D converter 115 to make initial settings. Meanwhile, the mainmirror 105 flips up, the shutter drive circuit 111 opens the shutter110, and the subject image is incident on the image sensor 112 throughthe shooting lens 101.

The signal from the image sensor 112 is converted to a digital signal bythe A/D converter 115, developed by the developing circuit 116 k of theimage signal processing circuit 116, and converted into a suitable imageby the luminance adjustment circuit 116 a and the gamma correctioncircuit 116 b. This image data is then converted by the magnificationcircuit 116 f to an image size suited to the display member 118, and isthen displayed. What is known as a “live view display” is achieved byrepeating this process 24 to 60 times per second.

The user adjusts the shooting direction, angle of view, and the likewhile confirming the live view display, and presses the switch SW1(125). The exposure amount is calculated upon the switch SW1 (125) beingpressed. If live view shooting is not being used, the light reflected bythe sub mirror 106 is received by the exposure amount calculationcircuit 109, which then calculates an appropriate exposure amount. Iflive view shooting is being used, the appropriate exposure amount isdetermined by an exposure amount calculation circuit (not shown)included in the image signal processing circuit 116. Then, themicrocomputer 123 drives the aperture stop 103 using the aperture drivecircuit 104, controls the sensitivity, accumulation time, and the likeof the image sensor 112, and so on. A program chart for ensuring anexposure time at which stars will not appear as lines is used whenshooting a starry sky panorama. On the other hand, the AF drive circuit102 drives the shooting lens 101 to adjust the focus. Once the shootingpreparations have ended, the user is notified using a buzzer or the like(not shown). The user then points the image capturing device in thedirection he/she wishes to start shooting from, and presses the switchSW2 (126), whereupon the shooting of a starry sky panorama is started.

The shooting of a starry sky panorama will be described in furtherdetail next using the flowcharts in FIGS. 4A and 4B and the data flowdiagrams in FIGS. 5A and 5B.

When the shooting of a starry sky panorama is started, first, themicrocomputer 123 acquires lens information in step S401. This lensinformation includes data for correcting distortion, a drop in theamount of light in the peripheral parts of the lens, and the like(described later).

In step S402, long-exposure shooting is carried out for the first image.The image sensor 112 and the A/D converter 115 are set for live viewdriving, and thus the driving is switched to driving for shooting astill image. The aperture stop 103 is adjusted to the exposure amountdetermined earlier, and the focal plane shutter 110 is opened and closedto expose the image sensor 112. The image signal obtained by the imagesensor 112 is converted to a digital signal by the A/D converter 115 andstored in the buffer memory 122. This image data is subjected toprocessing such as shading correction by a circuit (not shown) includedin the image signal processing circuit 116. Image data that hasundergone the minimum amount of processing in this manner is called RAWimage data 501. This RAW image data 501 is developed by the developingcircuit 116 k to obtain YUV image data 502.

Next, in step S403, high-sensitivity short-exposure shooting is carriedout for the first image to obtain a short-exposure image.High-sensitivity short-exposure shooting carried out immediately beforethe long-exposure shooting will be referred to as A, andhigh-sensitivity short-exposure shooting carried out immediately afterthe long-exposure shooting will be referred to as B. Thehigh-sensitivity short-exposure shooting is used only to calculate amovement amount, and is not used to obtain images for panoramiccombination. The high-sensitivity short-exposure shooting B is carriedout in step S403. The stars will be small and faint in short-exposureshooting, and thus the shooting is carried out at a higher ISOsensitivity. As with the long-exposure shooting, RAW image data 504 isdeveloped by the developing circuit 116 k to obtain YUV image data 505.

In step S404, the gyrosensor 133 is first reset at the point in time ofthe first image, to make it possible to later obtain the extent to whichthe image capturing device 100 has swung (pivoted) leading up to theshooting of the second image, in step S409.

In step S405, the geometric conversion circuit 116 e corrects thedeveloped image data 502 and 505 from the long-exposure shooting and theshort-exposure shooting, respectively, for distortion produced by theshooting lens 101, using a known technique, to obtaindistortion-corrected image data 503 and 506. The long-exposuredistortion-corrected image data 503 is reduced by the magnificationcircuit 116 f in accordance with the number of pixels in the liquidcrystal monitor to display the data in the display member 118, and isthen stored in VRAM 511.

Next, in step S406, the high-sensitivity short-exposure shooting A forthe second image is carried out to obtain image data 508. In step S407,long-exposure shooting is carried out for the second image to obtainimage data 513. Furthermore, in step S408, the high-sensitivityshort-exposure shooting B is carried out for the second image to obtainimage data 516. As in step S402, the RAW image data 508, 513, and 516 isdeveloped by the developing circuit 116 k to obtain YUV image data 509,514, and 517.

In step S409, gyrosensor information, which is a detection value fromthe gyrosensor 133, is obtained in order to obtain the amount by whichthe image capturing device 100 has swung since the previous shooting.Although values in two axial directions of the image capturing device,namely the yaw direction and the pitch direction, are obtained as thegyrosensor information, it is preferable that values be obtained for athird axial direction, namely the roll direction corresponding torotation about the optical axis, as well. Although the outputs from thegyrosensor 133 are themselves angular velocities, panoramic shootingrequires the extent to which the apparatus has swung since the previousshooting. Thus the angular velocities from the previous shooting to thenext shooting are integrated, and a rotation angle 507 from the previousshooting is calculated and stored for the second and subsequent images.

In step S410, the rotation angle 507 is converted into pixel units onthe basis of the focal length and angle of view of the lens obtained instep S401, information of the image sensor, and so on.

Assuming an effective focal length of f [mm] and an image sensor widthof w [mm], the angle of view (a) of a typical lens having no distortionor the distortion-corrected angle of view (a) is calculated through thefollowing Formula 1.α[°]=2×arctan(w[mm]÷2÷f[mm])  (Formula 1)

Assuming the size of the image sensor per pixel is p [μm] and the swingangle [° ] is 0, a movement amount d [pix] in the image is calculatedthrough Formula 2.d[pix]=tan(α[°]÷2)×f[mm]/p[μm]×1000  (Formula 2)In step S411, the data for the second image is subjected to distortioncorrection in the same manner as the distortion correction for the firstimages (step S405), to obtain distortion-corrected image data 510, 515,and 518. As with the first image, the distortion-corrected long-exposureshooting image data 515 is reduced by the magnification circuit 116 f inaccordance with the number of pixels in the liquid crystal monitor todisplay the data in the display member 118, and is then stored in VRAM519.

In step S412, the movement amount calculation circuit 116 c is used tocalculate a movement amount from the image data 506 obtained from thehigh-sensitivity short-exposure shooting B for the first image and theimage data 510 obtained from the high-sensitivity short-exposureshooting A for the second image. A known method can be used to detectthe movement amount, as described above. However, in the presentembodiment, the movement amount detection circuit 116 c finds andsamples several feature points within the image to calculate an affinecoefficient 512.

Specifically, edges are detected, feature points are extracted, and themovement amount is calculated. Here, assume that feature point 1 hasmoved from coordinates (x1,y1) to coordinates (u1,v1), feature point 2has moved from coordinates (x2,y2) to coordinates (u2,v2), and featurepoint 3 has moved from coordinates (x3,y3) to coordinates (u3,v3), forexample. In this case, Formulas 3 and 4 are obtained by creatingsimultaneous equations from Formula 1.

$\begin{matrix}{{\begin{pmatrix}{x\; 1} & {y\; 1} & 1 \\{x\; 2} & {y\; 2} & 1 \\{x\; 3} & {y\; 3} & 1\end{pmatrix}\begin{pmatrix}a \\b \\c\end{pmatrix}} = \begin{pmatrix}{u\; 1} \\{u\; 2} \\{u\; 3}\end{pmatrix}} & {{Formula}\mspace{14mu} 3} \\{{\begin{pmatrix}{x\; 1} & {y\; 1} & 1 \\{x\; 2} & {y\; 2} & 1 \\{x\; 3} & {y\; 3} & 1\end{pmatrix}\begin{pmatrix}d \\e \\f\end{pmatrix}} = \begin{pmatrix}{v\; 1} \\{v\; 2} \\{v\; 3}\end{pmatrix}} & {{Formula}\mspace{14mu} 4}\end{matrix}$

Solving these equations makes it possible to calculate the affinecoefficients a to f. If four or more feature points have beensuccessfully detected, nearby points are excluded, and the points arenormalized using the least-squares method. If three points cannot befound, or the extracted three points are linear in form and two of thethree points are nearby, it is determined that the movement amountcalculation has failed.

If the movement amount (affine coefficient) calculated from the imagesin this manner differs greatly from the movement amount based on therotation angle 507 calculated from the values detected by the gyrosensorin step S410, it is conceivable that a repeating pattern or a movingobject is present in the images. In this case, various measures areconceivable, such as calculating the movement amount again underdifferent conditions, assuming the shot has failed and returning theprocess to the next shooting (step S406), or providing a warning thatthe starry sky panoramic shooting has failed.

In step S413, the images obtained from the long-exposure shooting insteps S402 and S407 are positioned using the positioning circuit 116 d,on the basis of the movement amount (affine coefficient) calculated fromthe images, and positioned image data 621 is obtained.

In step S414, the image data 620 from the first image and the positionedimage data 621 from the second image are combined using the combiningcircuit 116 j to obtain combined image data 622. Note that carrying outprocessing on the Nth image (where N>2), the positioned image data 621from the Nth image is combined with the results of the combinationcarried out thus far, i.e., the combined image data 620 up to the(N−1)th image.

In step S415, if the switch SW2 (126) is depressed, the process returnsto the next shooting in step S406, whereas if the switch SW2 (126) isnot depressed, the process moves to step S416. In step S416, the imagedata is compressed according to a generic format such as JPEG using thecompression/decompression circuit 116 l, and in step S417, thecompressed data is saved in the memory 120.

Note that at this time, it is preferable that y correction be carriedout by the gamma correction circuit 116 b, and that correction becarried out to make the overall color tone of the image uniform, to makeit easier to see dark parts in the combined image. Furthermore, becausethe image obtained as a result is large, the magnification circuit 116 fmay change the size of the image to a size designated in advance by theuser. Furthermore, it is preferable that a maximum inscribed rectangleor a predetermined region first be cut out by the trimming circuit 116 ebefore being saved.

Although the foregoing describes an example of shooting a plurality ofimages while moving the image capturing device in the horizontaldirection, the same method can be used when moving image capturingdevice in the vertical direction.

As described thus far, even if the stars, which serve as a subject, havemoved between images shot from different directions, a high-qualitypanoramic combination image can be shot with correct positioning andwithout increasing the sensitivity.

Second Embodiment

The present embodiment describes an example in which thehigh-sensitivity short-exposure shooting for calculating the movementamount is unnecessary, depending on the shooting conditions,environment, and the like for the starry sky panoramic shooting. FIGS.6A and 6B are flowcharts illustrating panoramic shooting operationsaccording to the second embodiment.

The processes of steps S601 to S602 from the start of the starry skypanoramic shooting correspond to the processes of steps S401 to S402 ofthe first embodiment, the processes of steps S604 to S606, to theprocesses of steps S403 to S405; the processes of steps S608 to S609, tothe processes of steps S406 to S407; and the processes of steps S611 toS620, to the processes of steps S408 to S417. These processes thereforewill not be described.

In step S603, it is determined whether or not it is necessary to carryout the high-sensitivity short-exposure shooting B after thelong-exposure shooting for the first image (step S602). Thedetermination is carried out as follows, for example. First, themicrocomputer 123 obtains the direction of the image capturing device asdetected by the azimuth indicator 134, and calculates the amount ofmovement of the stars between the shots. If it is determined that thestars have not moved, the process of step S605 is carried out withoutcarrying out the high-sensitivity short-exposure shooting B in stepS604. The determination to carry out the high-sensitivity short-exposureshooting B may be made using settings such as the accumulation time.

Whether or not it is necessary to carry out the high-sensitivityshort-exposure shooting A and B before and after the long-exposureshooting (step S609) for the second and subsequent images is determinedin steps S607 and S610, through the same process as that used in stepS603. If it is determined that the stars have not moved, the process ofsteps S609 and S612 are carried out without carrying out thehigh-sensitivity short-exposure shooting A in step S608 and thehigh-sensitivity short-exposure shooting B in step S611.

As described thus far, determining the shooting conditions, shootingenvironment, and the like for the starry sky panoramic shooting makes itpossible to omit high-sensitivity short-exposure shooting not necessaryfor the positioning. This makes it possible to reduce the power consumedfor shooting. Although the present embodiment describes a case where thedetermination is made before all instances of high-sensitivityshort-exposure shooting, the configuration may be such that thedetermination is made only once, if, while the image capturing device isin a standby state, it can be determined that all instances of thehigh-sensitivity short-exposure shooting are unnecessary.

Third Embodiment

The present embodiment describes, with reference to FIGS. 4A, 4B, 7A,and 7B, an example of displaying a suitable warning to the user when itis conceivable that the positioning will fail in the starry skypanoramic shooting.

When the high-sensitivity short-exposure shooting B in 504 and thehigh-sensitivity short-exposure shooting A in 508 are carried out, themicrocomputer 123 obtains and stores the time from the internal clock132. The microcomputer 123 calculates interval between to the twoshooting times. If the interval is greater than or equal to a set time,it is determined that the stars have moved too much, and the warningscreen illustrated in FIG. 7A is displayed.

The microcomputer 123 also detects the amount by which the imagecapturing device has swung from the rotation angle 507, which is thegyrosensor information obtained in step S409. If the amount exceeds aset swing amount, it is determined that positioning cannot be carriedout, and the warning screen illustrated in FIG. 7B is displayed.

As described thus far, the convenience can be enhanced for the user bydisplaying a warning in advance in a situation where positioning isestimated to be impossible in the starry sky panoramic shooting.

Fourth Embodiment

A starry sky panoramic shooting process executed by the microcomputer123 will be described next using the flowchart in FIG. 9. Although thestarry sky panorama shooting mode includes a mode that shoots whilechanging the direction of the image capturing device 100 in thehorizontal direction and a mode that shoots while changing the directionof the image capturing device 100 in the vertical direction, the formerwill be described here.

When the user selects the starry sky panorama shooting mode by operatingthe menu button (YES in step S900), shooting preparations are made (stepS901).

Here, “shooting preparations” indicate the following specific processes,i.e., supplying power from the power source unit 131 to the image sensor112, the A/D converter 115, and the like, and resetting those units.Next, the mirror drive circuit 107 is driven to retract the main mirror105 from the light beam, the shutter drive circuit 111 is driven to openthe shutter 110, and the subject image is formed on the image sensor 112through the shooting lens 101.

The live view display is then started in the liquid crystal monitor 128.In other words, the image signal from the image sensor 112 is convertedinto a digital signal by the A/D converter 115, the developing circuit116 k of the image signal processing circuit 116 develops the digitalsignal into image data, and the brightness and luminance of the imageare then adjusted by the brightness adjustment circuit 116 a and thegamma correction circuit 116 b. Furthermore, the image data is convertedto an image size suited to the liquid crystal monitor 128 by themagnification circuit 116 f, and is then displayed. This is repeated 24to 60 times per second.

Next, the user adjusts the angle of view while confirming the live viewin the liquid crystal monitor 128. When the user then presses therelease switch 125 halfway, the release switch 125 turns a SW1 signalon. When the SW1 signal turns on, the microcomputer 123 carries outphotometry operations, and the exposure amount calculation circuit 109calculates the exposure amount. In the present embodiment, the live viewis suspended when calculating the exposure amount, and the lightreflected by the sub mirror 106 is conducted to a sensor within theexposure amount calculation circuit 109. The exposure amount calculationcircuit 109 calculates the optimal exposure amount. Note that the liveview may be continued while calculating the exposure amount. In thiscase, the optimal exposure amount is determined by an exposure amountcalculation circuit (not shown) included in the image signal processingcircuit 116.

Then, exposure control is carried out on the basis of the calculatedexposure amount. Specifically, the aperture value is determined on thebasis of the calculated exposure amount, and the aperture value iscommunicated to the aperture drive circuit 104, which then drives theaperture stop 103 to that aperture value. The sensitivity, accumulationtime, and the like of the image sensor 112 are also controlled on thebasis of the calculated exposure amount. At this time, the accumulationtime is set using a program chart for ensuring an exposure time at whichstars will not appear as lines is used in the long-exposure shootingduring the starry sky panoramic shooting.

After the exposure control, the AF drive circuit 102 drives the shootinglens 101 to adjust the focus. When this is complete, the user isnotified that the starry sky panoramic shooting preparations arecomplete using a buzzer or the like (not shown), which ends the shootingpreparations.

When the shooting preparations in step S901 are complete and the userhas received the aforementioned notification, the user points the imagecapturing device 100 in the direction he/she wishes to start theshooting from, and fully depresses the release switch 125. The releaseswitch 125 turns an SW2 signal on. When the SW1 signal turns on (YES instep S902), the microcomputer 123 transitions to parallel processing forshooting and generating a panoramic image of only the background (stepS903). In this process, a panoramic image of only the background isgenerated in parallel with the shooting for obtaining all the imagesnecessary to generate the panoramic starry sky image. The method forgenerating a panoramic image of only the background will be described indetail hereinafter.

The method for generating a panoramic image of only the background isalmost the same as the method illustrated in the flowchart of FIGS. 4Aand 4B, but is different in that a comparative dark combination iscarried out in step S414, after which the process ends. This will bedescribed next.

In step S414 of FIG. 4B, the combining circuit 116 j carries out acomparative dark combination on the positioned image 521 obtained instep S413 and the combined image 520 resulting from the processing up tothe N−1th image (a comparative dark combination image, in the presentembodiment), and obtains a new comparative dark combination image 622.If N=2, a geometrically-converted image 503 obtained from thelong-exposure shooting for the first image is used as the comparativedark combined image 520 resulting from the combination up to the N−1thimage. The comparative dark combination process will be described nextusing the conceptual diagram for an image processing, illustrated inFIG. 10.

As illustrated in FIG. 10, first to fourth geometrically-convertedimages 1005, 1008, 1011, and 1014 (called simply “long-exposure shootingimages” hereinafter), which are obtained from long-exposure shooting atangles of view 1001 to 1004, are obtained while the stars, which are thesubject, are moving.

Meanwhile, a geometrically-converted image 1006 (called a“short-exposure shooting B image” (second short-exposure shooting image)hereinafter) is obtained from the short-exposure shooting B for thefirst image immediately after the long-exposure shooting for the firstimage. A geometrically-converted image 1007 (called a “short-exposureshooting A image” (first short-exposure shooting image) hereinafter) isobtained from the short-exposure shooting A immediately before thelong-exposure shooting for the second image, and a short-exposureshooting B image 1009 is obtained immediately after the long-exposureshooting for the second image. In the same manner, short-exposureshooting A images 1010 and 1013 are obtained immediately before thelong-exposure shooting for the third and fourth images, andshort-exposure shooting B images 1012 and 1015 are obtained immediatelyafter the long-exposure shooting for the third and fourth images.

First, a movement amount 1016 of the stars, in an overlapping regionbetween the short-exposure shooting B image 1006 of the first image andthe short-exposure shooting A image 1007 of the second image, iscalculated. Likewise, a movement amount 1017 is calculated using theshort-exposure shooting B image 1009 of the second image and theshort-exposure shooting A image 1010 of the third image, and a movementamount 1018 is calculated using the short-exposure shooting B image 1012of the third image and the short-exposure shooting A image 1013 of thefourth image.

The calculated movement amounts 1016, 1017, and 1018 are used whenobtaining comparative dark combination images 1019, 1020, and 1021 bycarrying out the comparative dark combination process on thelong-exposure shooting images 1008, 1011, and 1014. Specifically, thecomparative dark combination image 1019 is generated by comparative darkcombination, in a state where the long-exposure shooting image 1001 ofthe first image and the long-exposure shooting image 1008 of the secondimage have been positioned on the basis of the movement amount 1016. Asindicated by the comparative dark combination image 1019, in theoverlapping regions of the images subject to the comparative darkcombination (the long-exposure shooting images 1005 and 1008, here), thebackground is stationary and therefore remains, but the stars are movingand therefore do not remain.

Note that when N>2, the comparative dark combination image resultingfrom the combination up to the N−1th image and the long-exposureshooting image of the Nth image are subject to the comparative darkcombination in a state where those images have been positioned on thebasis of the movement amount calculated using the short-exposureshooting B image of the N−1th image and the short-exposure shooting Aimage of the Nth image. A comparative dark combination image that is theresult of the combination up to the Nth image is generated as a result.For example, the comparative dark combination image 1020, which is theresult of the combination up to the third image, is generated bycomparative dark combination, in a state where the comparative darkcombination image 919, which is the result of the combination up to thesecond image, and the long-exposure shooting image 911 of the thirdimage, have been positioned on the basis of the movement amount 917.Repeating the same comparative dark combination makes it possible togenerate a panoramic image of only the background, in which the regionsof the background area are gradually connected together, as indicated bythe comparative dark combination image 1021.

Returning to FIG. 4B, in step S415, the release switch 125 turns the SW2signal on upon the user fully depressing the release switch 125 during aperiod from when the short-exposure shooting B of step S408 has ended towhen a predetermined amount of time has passed. In this case, it isdetermined that shooting has not yet ended (NO in step S415), the countN is incremented by 1, and the process is repeated from step S406. Onthe other hand, if the user has not fully depressed the release switch125 during the stated period, it is determined that the shooting hasended (YES in step S415), and the process ends. A case where the valueof the counter N is n when it is determined, in step S415, that theshooting has ended, will be described below.

According to the processing of the present embodiment as described thusfar, by incrementing the counter N from 2 to n, long-exposure shootingis carried out at each angle of view, and short-exposure shooting iscarried out before and after each instance of long-exposure shooting.Positioning and comparative dark combination are repeated for eachlong-exposure shooting image with the movement amount calculated fromthe obtained short-exposure shooting images. This makes it possible togenerate a panoramic image of only the background, in which the regionsof the background area are gradually connected together, as indicated bythe comparative dark combination image 1021.

Additionally, as indicated by steps S402 and S403, the short-exposureshooting is carried out only after the long-exposure shooting for thefirst angle of view, i.e., when obtaining the long-exposure shootingimage for the first image.

Returning to FIG. 9, when the parallel processing for shooting andgenerating a panoramic image of only the background of step S903,described in detail with reference to FIGS. 4A and 4B, has ended, theprocess transitions to a process for generating a panorama of only thestars (step S904).

The process for generating a panoramic image of only the stars in stepS904 will be described in detail hereinafter using the flowchart in FIG.11 and the conceptual diagram of image processing in FIG. 10.

In FIG. 11, first, the counter N is reset to 1 (step S1101).

Next, a differential image extraction circuit 116 m carries outdifferential image extraction on the long-exposure shooting image fromthe Nth image and the comparative dark combination image that is theresult of the combination up to the N+1th image, and a comparative darkcombination image for the Nth image is generated (step S1102).

For example, as illustrated in FIG. 10, if N=1, the differential imageextraction circuit 116 m carries out the differential image extractionon the long-exposure shooting image 1005 of the first image and thecomparative dark combination image 1019 that is the result of thecombination up to the second image, and an image 1022 of only the starsis generated for the first image. The image 1022 of only the stars is animage in which only the stars in the overlapping region of thelong-exposure shooting image 1005 remain as a differential image.

The value of the counter N is then incremented (step S1103).

The processing from step S1102 is then repeated until the long-exposureshooting image used in step S1102 reaches N>n, i.e., until the N−1thimage is the long-exposure shooting image of the nth image (the lastimage) (NO in step S1104). As a result, an image 1023 of only the starsfor the second image is generated when N=2, and an image 1024 of onlythe stars for the third image is generated when N=3, as illustrated inFIG. 10.

When the final long-exposure shooting image is determined to have beenreached in step S1104, the process moves to step S1105, where thecounter N is reset to 2.

Next, the movement amount calculation circuit 116 c calculates amovement amount of the stars between the images of only the stars fromthe N−1th image and the Nth image (a movement amount between images ofonly the stars) (step S1106). For example, when N=2, the movement amountcalculation circuit 116 c calculates the movement amount from the images1022 and 1023 of only the stars, of the first image and the secondimage, as illustrated in FIG. 10.

Next, the positioning circuit 116 d positions the images of only thestars, of the N−1th image and the Nth image, on the basis of themovement amount calculated in step S1106 (step S1107). For example, ifN=2, the images 1022 and 1023 of only the stars, of the first image andthe second image, are positioned, as illustrated in FIG. 10.

In step S1108, the combining circuit 116 j carries out a comparativelight combination on a comparative light combined image, which is theresult of combining the images of only the stars up to the N−1th image,positioned in step S1107, and the image of only the stars from the Nthimage. A comparative light combined image, which is the result of thecombination up to the Nth image, is generated as a result. For example,as illustrated in FIG. 10, if N=3, the combining circuit 116 j carriesout a comparative light combination on a comparative light combinedimage 1025, which is the result of the combination up to the secondimage, and the image 1024 of only the stars, from the third image. Acomparative light combined image 1026, which is the result of thecombination up to the third image, is generated as a result. Note thatthe image 1022 of only the stars of the first image is used as thecomparative light combined image, which is the result of the combinationup to the N−1th image, only when N=2.

Next, the value of the counter N is incremented (step S1109), and theprocessing from step S1106 is then repeated until N>n, i.e., until theN−1th image is the image of only the stars in the nth image (the lastimage) (NO in step S1110), after which the process ends.

As described above, according to the process of FIG. 11, by incrementingthe value of the counter N in order from 2 to n, the positioning andcomparative light combination for each movement amount between theimages of only the stars is repeated for each image of only starsextracted from the differences between each long-exposure shooting imageand the comparative dark combination result. This makes it possible togenerate a panoramic image of only the stars, in which the regions ofthe starry areas are gradually connected together, as indicated by thecomparative light combined image 1026.

Returning to FIG. 9, when the panoramic image generation process foronly the stars in step S904, described in detail with reference to FIG.12, is completed, the process transitions to the panoramic starry skyimage generation process (step S905).

The panoramic starry sky image generation process of step S905 will bedescribed in detail hereinafter using the conceptual diagram of imageprocessing in FIG. 10 and the flowchart in FIG. 12.

In FIG. 12, first, a panoramic image of only the background (e.g., thecomparative dark combination image 1021 in FIG. 10) and a panoramicimage of only the stars (e.g., the comparative light combined image 1026in FIG. 10) are added and combined by the combining circuit 116 j (stepS1201). The panoramic starry sky image 1027 is generated as a result, asindicated in FIG. 10.

The generated panoramic starry sky image 1027 is compressed into ageneric format such as JPEG by the compression/decompression circuit 116l (step S1202) and stored in the memory 120 (step S1203), after whichthe processing ends. Note that at this time, it is preferable that ycorrection be carried out by the gamma correction circuit 116 b, andthat correction be carried out to make the overall color tone of theimage uniform, to make it easier to see dark parts in panoramic starrysky image 1027. Furthermore, because the image obtained as a result islarge, the magnification circuit 116 f may change the size of the imageto a size designated in advance by the user. Further still, it ispreferable that a maximum inscribed rectangle or a predetermined regionfirst be cut out by the trimming circuit 116 e before being saved.

Returning to FIG. 9, when the panoramic starry sky image generationprocess of step S905, described in detail with reference to FIG. 12 hasended, the overall starry sky panoramic shooting process ends.

Although the present embodiment describes an example in which the imagecapturing device 100 is swung in the horizontal direction, the imagecapturing device 100 may be swung in the vertical direction as well.

As described above, in starry sky panoramic shooting, even if the starsserving as the subject move between instances of shooting at differentangles of view, both the background and the stars can be positioned.

Fifth Embodiment

In the present embodiment, in the process of generating a panoramicimage of only the stars carried out in step S904 of the starry skypanoramic shooting process illustrated in FIG. 9, an appropriate warningis displayed to the user when the calculation of the movement amountbetween images of only the starts fails, and replacement means for thecomparative dark combination process are carried out. The presentembodiment will be described in detail below with reference to FIGS. 13,14, and 15.

The present embodiment differs from the first embodiment only in termsof part of the panoramic image generation process for only the stars,but the other processing and hardware configurations are the same as inthe first embodiment. As such, like configurations and steps are giventhe same reference numerals, and redundant descriptions will be omitted.

In FIG. 13, when the processes of steps S1101 to S1106 of FIG. 11 havebeen performed, the process moves to step S1101.

In step S1101, it is determined whether the movement amount between theimages of only the stars has been successfully calculated, andspecifically, whether or not feature points have been successfullyextracted in the overlapping region in the images 1401 and 1402 of onlythe stars, for M−1th and Mth images indicated in FIG. 14 (where M is aninteger equal to 2≤M≤n). If the result of this determination indicatessuccess, the processing from step S1107 in FIG. 11 is carried out, afterwhich this process ends.

On the other hand, if the feature point extraction has failed (NO instep S1101), a warning is displayed for the user, indicating that thepositioning of the images of only the stars has failed (step S1311). Themethod for making this warning display is not particularly limited, butfor example, the notification screen shown in FIG. 15 is displayed inthe liquid crystal monitor 128.

Next, the combining circuit 116 j generates a combined image 1405 theimages 1401 and 1402 of only the stars, from the M−1th and Mth images(step S1312). At this time, in the present embodiment, the image 1401 ofonly the stars, from the M−1th image, is employed as the image of theoverlapping region between the images 1401 and 1402 of only the stars,but the image 1402 of only the stars, from the Mth image, may beemployed instead. Also, because luminance differences arise easily atthe boundary areas of such a combined image 1405, a filter may beapplied to the boundary areas to add blur or the like.

Thereafter, the processing from step S1109 and on is carried out, andthe overall process ends.

As described above, when the feature point extraction fails whencalculating the movement amount between the images of only the stars, aprocess for combining both of the images is carried out. This makes itpossible to prevent a situation in which the panoramic starry sky imagecannot be generated despite the user taking multiple shots over a periodof time in the parallel processing for shooting and generating apanoramic image of only the background, illustrated in FIGS. 6A and 6B.

On the other hand, when carrying out such a combining process, it isconceivable that the positioning of the images of only the stars willfail in the starry sky panoramic shooting. Accordingly, a panoramicstarry sky image is generated while displaying a warning in advanceimmediately before performing the combining process, which makes itpossible to improve the convenience for the user.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Applications No.2018-153205, filed Aug. 16, 2018, and No. 2018-167870, filed Sep. 7,2018 which are hereby incorporated by reference herein in theirentirety.

What is claimed is:
 1. An image capturing device capable of generating asingle combined image by combining a plurality of images, each imagebeing from a different image capturing region, and each image having aregion at least partially shared by another image capturing region, thedevice comprising: an image sensor configured to capture a subjectimage, the image sensor: capturing a first image in a first imagecapturing region; capturing a second image in the first image capturingregion at a shorter exposure time than that of the first image, aftercapturing the first image, the second image being captured in the sameimage capturing region as the first image; capturing a third image in asecond image capturing region at a shorter exposure time than that ofthe first image, after capturing the second image; and capturing afourth image in the second image capturing region at a longer exposuretime than those of the second and third images, after capturing thethird image, the fourth image being captured in the same image capturingregion as the third image; and the device further comprising: at leastone processor or circuit configured to function as: a combining unitconfigured to carry out alignment processing between the images usingthe second image and the third image, and combine the first image andthe fourth image on the basis of a result of the alignment processing, asecond determination unit configured to determine whether or not thefirst image and the fourth image can be aligned, and warning unitconfigured to issue a warning when the second determination unitdetermines that the first image and the fourth image cannot be aligned,wherein the subject of the capturing is a starry sky, and the firstimage and the fourth image are images shot at long exposures forshooting the starry sky.
 2. The image capturing device according toclaim 1, wherein the second image and the third image are images shotwith the sensitivity of the image sensor having been increased more thanthe first image and the fourth image.
 3. The image capturing deviceaccording to claim 1, wherein the at least one processor or circuit isconfigured to further function as: a first determination unit configuredto determine whether or not it is necessary to capture the second image,wherein when the first determination unit determines that it is notnecessary to capture the second image, the image sensor omits thecapturing of the second image.
 4. The image capturing device accordingto claim 1, wherein the at least one processor or circuit is configuredto further function as: a setting unit configured to transition to amode for shooting a starry sky, wherein the combining unit carries outthe combining in the mode for shooting the starry sky.
 5. The imagecapturing device according to claim 1, wherein the second determinationunit determines that the first image and the fourth image cannot bealigned when a movement amount of the subject between the second imageand the third image is greater than a predetermined amount.
 6. An imagecapturing device capable of generating a single combined image bycombining a plurality of images, each image being from a different imagecapturing region, and each image having a region at least partiallyshared by another image capturing region, the device comprising: animage sensor configured to capture a subject image, the image sensor:capturing a first image in a first image capturing region; capturing asecond image in the first image capturing region at a shorter exposuretime than that of the first image, after capturing the first image, thesecond image being captured in the same image capturing region as thefirst image; capturing a third image in a second image capturing regionat a shorter exposure time than that of the first image, after capturingthe second image; and capturing a fourth image in the second imagecapturing region at a longer exposure time than those of the second andthird images, after capturing the third image, the fourth image beingcaptured in the same image capturing region as the third image; and thedevice further comprising: at least one processor or circuit configuredto function as: a combining unit configured to carry out alignmentprocessing between the images using the second image and the thirdimage, and combine the first image and the fourth image on the basis ofa result of the alignment processing, a second determination unitconfigured to determine whether or not the first image and the fourthimage can be aligned, and a time measurement unit configured to measurethe amount of time that has passed between the capturing of the secondimage and the capturing of the third image, wherein the subject of thecapturing is a starry sky, and the first image and the fourth image areimages shot at long exposures for shooting the starry sky, and whereinthe second determination unit determines that the first image and thefourth image cannot be aligned when the amount of time that has passedis greater than a predetermined amount of time.
 7. An image processingdevice capable of generating a single combined image by combining aplurality of images, each image being from a different image capturingregion, and each image having a region at least partially shared byanother image capturing region, the device comprising: at least oneprocessor or circuit configured to function as: an obtainment unitconfigured to obtain a first image captured in a first image capturingregion, a second image captured in the first image capturing region at ashorter exposure time than that of the first image after the first imagehas been captured, a third image captured in a second image capturingregion at a shorter exposure time than that of the first image after thesecond image has been captured, and a fourth image captured in thesecond image capturing region at a longer exposure time than those ofthe second and third images after the third image has been captured, theimages being captured by image sensor; and a combining unit configuredto carry out alignment processing between the images using the secondimage and the third image, and combine the first image and the fourthimage on the basis of a result of the alignment processing, adetermination unit configured to determine whether or not the firstimage and the fourth image can be aligned, and a warning unit configuredto issue a warning when the determination unit determines that the firstimage and the fourth image cannot be aligned, wherein the subject of thecapturing is a starry sky, and the first image and the fourth image areimages shot at long exposures for shooting the starry sky, and whereinthe second image is captured in the same image capturing region as thefirst image, and the fourth image is captured in the same imagecapturing region as the third image.
 8. The image processing deviceaccording to claim 7, wherein the second image and the third image areimages shot with the sensitivity of the image sensor having beenincreased more than the first image and the fourth image.
 9. The imageprocessing device according to claim 7, wherein the at least oneprocessor or circuit is configured to further function as: a calculationunit configured to calculate a movement amount of the subject betweenthe second image and the third image, wherein the determination unitdetermines that the first image and the fourth image cannot be alignedwhen the movement amount of the subject is greater than a predeterminedamount.
 10. An image processing device capable of generating a singlecombined image by combining a plurality of images, each image being froma different image capturing region, and each image having a region atleast partially shared by another image capturing region, the devicecomprising: at least one processor or circuit configured to function as:an obtainment unit configured to obtain a first image captured in afirst image capturing region, a second image captured in the first imagecapturing region at a shorter exposure time than that of the first imageafter the first image has been captured, a third image captured in asecond image capturing region at a shorter exposure time than that ofthe first image after the second image has been captured, and a fourthimage captured in the second image capturing region at a longer exposuretime than those of the second and third images after the third image hasbeen captured, the images being captured by image sensor; and acombining unit configured to carry out alignment processing between theimages using the second image and the third image, and combine the firstimage and the fourth image on the basis of a result of the alignmentprocessing, a determination unit configured to determine whether or notthe first image and the fourth image can be aligned, and a timemeasurement unit configured to measure the amount of time that haspassed between the capturing of the second image and the capturing ofthe third image, wherein the subject of the capturing is a starry sky,and the first image and the fourth image are images shot at longexposures for shooting the starry sky, wherein the second image iscaptured in the same image capturing region as the first image, and thefourth image is captured in the same image capturing region as the thirdimage, and wherein the determination unit determines that the firstimage and the fourth image cannot be aligned when the amount of timethat has passed is greater than a predetermined amount of time.
 11. Amethod of controlling an image capturing device capable of generating asingle combined image by combining a plurality of images, each imagebeing from a different image capturing region, and each image having aregion at least partially shared by another image capturing region, themethod comprising: capturing a first image in a first image capturingregion; capturing a second image in the first image capturing region ata shorter exposure time than that of the first image, after capturingthe first image, the second image being captured in the same imagecapturing region as the first image; capturing a third image in a secondimage capturing region at a shorter exposure time than that of the firstimage, after capturing the second image; capturing a fourth image in thesecond image capturing region at a longer exposure time than those ofthe second and third images, after capturing the third image, the fourthimage being captured in the same image capturing region as the thirdimage; carrying out alignment processing between the images using thesecond image and the third image, and combining the first image and thefourth image on the basis of a result of the alignment processing,determining whether or not the first image and the fourth image can bealigned, and issuing a warning when the determining determines that thefirst image and the fourth image cannot be aligned, wherein the subjectof the capturing is a starry sky, and the first image and the fourthimage are images shot at long exposures for shooting the starry sky. 12.A method of controlling an image processing device capable of generatinga single combined image by combining a plurality of images, each imagebeing from a different image capturing region, and each image having aregion at least partially shared by another image capturing region, themethod comprising: obtaining a first image captured in a first imagecapturing region, a second image captured in the first image capturingregion at a shorter exposure time than that of the first image after thefirst image has been captured, a third image captured in a second imagecapturing region at a shorter exposure time than that of the first imageafter the second image has been captured, and a fourth image captured inthe second image capturing region at a longer exposure time than thoseof the second and third images after the third image has been captured,the images being captured by image sensor; and carrying out alignmentprocessing between the images using the second image and the thirdimage, and combining the first image and the fourth image on the basisof a result of the alignment processing, determining whether or not thefirst image and the fourth image can be aligned, and issuing a warningwhen the determining determines that the first image and the fourthimage cannot be aligned, wherein the subject of the capturing is astarry sky, and the first image and the fourth image are images shot atlong exposures for shooting the starry sky, and wherein the second imageis captured in the same image capturing region as the first image, andthe fourth image is captured in the same image capturing region as thethird image.
 13. A non-transitory computer-readable storage mediumstoring a program for causing a computer to execute the steps of amethod of controlling an image capturing device capable of generating asingle combined image by combining a plurality of images, each imagebeing from a different image capturing region, and each image having aregion at least partially shared by another image capturing region, themethod comprising: capturing a first image in a first image capturingregion; capturing a second image in the first image capturing region ata shorter exposure time than that of the first image, after capturingthe first image, the second image being captured in the same imagecapturing region as the first image; capturing a third image in a secondimage capturing region at a shorter exposure time than that of the firstimage, after capturing the second image; capturing a fourth image in thesecond image capturing region at a longer exposure time than those ofthe second and third images, after capturing the third image, the fourthimage being captured in the same image capturing region as the thirdimage; carrying out alignment processing between the images using thesecond image and the third image, and combining the first image and thefourth image on the basis of a result of the alignment processing,determining whether or not the first image and the fourth image can bealigned, and issuing a warning when the determining determines that thefirst image and the fourth image cannot be aligned, wherein the subjectof the capturing is a starry sky, and the first image and the fourthimage are images shot at long exposures for shooting the starry sky. 14.A non-transitory computer-readable storage medium storing a program forcausing a computer to execute the steps of a method of controlling animage processing device capable of generating a single combined image bycombining a plurality of images, each image being from a different imagecapturing region, and each image having a region at least partiallyshared by another image capturing region, the method comprising:obtaining a first image captured in a first image capturing region, asecond image captured in the first image capturing region at a shorterexposure time than that of the first image after the first image hasbeen captured, a third image captured in a second image capturing regionat a shorter exposure time than that of the first image after the secondimage has been captured, and a fourth image captured in the second imagecapturing region at a longer exposure time than those of the second andthird images after the third image has been captured, the images beingcaptured by image sensor; and carrying out alignment processing betweenthe images using the second image and the third image, and combining thefirst image and the fourth image on the basis of a result of thealignment processing, determining whether or not the first image and thefourth image can be aligned, and issuing a warning when the determiningdetermines that the first image and the fourth image cannot be aligned,wherein the subject of the capturing is a starry sky, and the firstimage and the fourth image are images shot at long exposures forshooting the starry sky, and wherein the second image is captured in thesame image capturing region as the first image, and the fourth image iscaptured in the same image capturing region as the third image.
 15. Theimage capturing device according to claim 1, wherein the first imagecapturing region and the second image capturing region are differentimage capturing regions.
 16. A method of controlling an image capturingdevice capable of generating a single combined image by combining aplurality of images, each image being from a different image capturingregion, and each image having a region at least partially shared byanother image capturing region, the method comprising: capturing a firstimage in a first image capturing region; capturing a second image in thefirst image capturing region at a shorter exposure time than that of thefirst image, after capturing the first image, the second image beingcaptured in the same image capturing region as the first image;capturing a third image in a second image capturing region at a shorterexposure time than that of the first image, after capturing the secondimage; capturing a fourth image in the second image capturing region ata longer exposure time than those of the second and third images, aftercapturing the third image, the fourth image being captured in the sameimage capturing region as the third image; carrying out alignmentprocessing between the images using the second image and the thirdimage, and combining the first image and the fourth image on the basisof a result of the alignment processing, determining whether or not thefirst image and the fourth image can be aligned, and measuring theamount of time that has passed between the capturing of the second imageand the capturing of the third image, wherein the subject of thecapturing is a starry sky, and the first image and the fourth image areimages shot at long exposures for shooting the starry sky, and whereinthe determining determines that the first image and the fourth imagecannot be aligned when the amount of time that has passed is greaterthan a predetermined amount of time.
 17. A method of controlling animage processing device capable of generating a single combined image bycombining a plurality of images, each image being from a different imagecapturing region, and each image having a region at least partiallyshared by another image capturing region, the method comprising:obtaining a first image captured in a first image capturing region, asecond image captured in the first image capturing region at a shorterexposure time than that of the first image after the first image hasbeen captured, a third image captured in a second image capturing regionat a shorter exposure time than that of the first image after the secondimage has been captured, and a fourth image captured in the second imagecapturing region at a longer exposure time than those of the second andthird images after the third image has been captured, the images beingcaptured by image sensor; and carrying out alignment processing betweenthe images using the second image and the third image, and combining thefirst image and the fourth image on the basis of a result of thealignment processing, determining whether or not the first image and thefourth image can be aligned, and measuring the amount of time that haspassed between the capturing of the second image and the capturing ofthe third image, wherein the subject of the capturing is a starry sky,and the first image and the fourth image are images shot at longexposures for shooting the starry sky, wherein the second image iscaptured in the same image capturing region as the first image, and thefourth image is captured in the same image capturing region as the thirdimage, and wherein the determining determines that the first image andthe fourth image cannot be aligned when the amount of time that haspassed is greater than a predetermined amount of time.
 18. Anon-transitory computer-readable storage medium storing a program forcausing a computer to execute the steps of a method of controlling animage capturing device capable of generating a single combined image bycombining a plurality of images, each image being from a different imagecapturing region, and each image having a region at least partiallyshared by another image capturing region, the method comprising:capturing a first image in a first image capturing region; capturing asecond image in the first image capturing region at a shorter exposuretime than that of the first image, after capturing the first image, thesecond image being captured in the same image capturing region as thefirst image; capturing a third image in a second image capturing regionat a shorter exposure time than that of the first image, after capturingthe second image; capturing a fourth image in the second image capturingregion at a longer exposure time than those of the second and thirdimages, after capturing the third image, the fourth image being capturedin the same image capturing region as the third image; carrying outalignment processing between the images using the second image and thethird image, and combining the first image and the fourth image on thebasis of a result of the alignment processing, determining whether ornot the first image and the fourth image can be aligned, and measuringthe amount of time that has passed between the capturing of the secondimage and the capturing of the third image, wherein the subject of thecapturing is a starry sky, and the first image and the fourth image areimages shot at long exposures for shooting the starry sky, and whereinthe determining determines that the first image and the fourth imagecannot be aligned when the amount of time that has passed is greaterthan a predetermined amount of time.
 19. A non-transitorycomputer-readable storage medium storing a program for causing acomputer to execute the steps of a method of controlling an imageprocessing device capable of generating a single combined image bycombining a plurality of images, each image being from a different imagecapturing region, and each image having a region at least partiallyshared by another image capturing region, the method comprising:obtaining a first image captured in a first image capturing region, asecond image captured in the first image capturing region at a shorterexposure time than that of the first image after the first image hasbeen captured, a third image captured in a second image capturing regionat a shorter exposure time than that of the first image after the secondimage has been captured, and a fourth image captured in the second imagecapturing region at a longer exposure time than those of the second andthird images after the third image has been captured, the images beingcaptured by image sensor; and carrying out alignment processing betweenthe images using the second image and the third image, and combining thefirst image and the fourth image on the basis of a result of thealignment processing, determining whether or not the first image and thefourth image can be aligned, and measuring the amount of time that haspassed between the capturing of the second image and the capturing ofthe third image, wherein the subject of the capturing is a starry sky,and the first image and the fourth image are images shot at longexposures for shooting the starry sky, wherein the second image iscaptured in the same image capturing region as the first image, and thefourth image is captured in the same image capturing region as the thirdimage, and wherein the determining determines that the first image andthe fourth image cannot be aligned when the amount of time that haspassed is greater than a predetermined amount of time.